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ABSTRACT 

Pronunciation is an important aubskill in second 
language learning, therefore worth evaluating. Its quality is 
commonly assessed in a global* impressionistic way by having learners 
read aloUd. While this allows Comparison of examinees' Skills* 
ability to read aloud is a possible confounding variable. An 
alternative method is to have learners read texts in which only 
marked elements are judged as correct or incorrect. A study compared 
the reliability and validity of this and the traditional, holistic 
method. Pronunciation tests were developed for Dutch learners of 
French and German that incorporated words in which pronunciation 
errors occur frequently* The tests were administered to secondary 
school students (Germanize, French«i9) . Recorded readings were rated 
in both the traditional way and with the marked-item method by 
teacher panels to measure reliability. Results suggest the new method 
can improve evaluation. To determine validity, another experiment 
simulated regular teacher evaluation of student pronunciation in 
French by comparing ratings of pronunciation int (1) spontaneous 
speech! (2) a traditional read-aloud text! and (3) marked items in a 
read-aloud text. These results suggest that (3) marked items in a 
read-aloud text. These results suggest the new alternative is not 
preferable to traditional, holistic rating because it can not give a 
general impression of pronunciation quality. Further research is 
recommended . ( KSE ) 
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ABSTRACT 



Pronunciation It regarded • valueble eubeklll In foreign language 
tSSSSn and u.tlnf. It. Quality to commonly <mm«*f tSJS 
ktl hfttHn» ftMtMi road tloudi A wort •yattmitfonl tno tranipirtnt 
epproech'to IviultSf I. proceed. The reWty of the experimental 
Spproeoh la compered with thVrelleblltty of e tredltlonel rating pro- 
cedure! In a Teeoond etudy the external velldity of the experiments 
approaoh la determined. 



1. Introduction 

There ere good reeeone for not neglecting pronunciation aa an important 
aubaktll in foreign language leeching end teeting. Flret of ell, with a good 
pronunciation one cen meke oneaelf better underetood. it givea onc'a oral 
production a certain redundency end tide cen help o epeeker to get hia 
moaeege acroaa more eerily. Secondly, pronuncietion ie quite attractive aa 
a learning objective beoauee of ite high pay-off. The number of eounda, 
round cluatora and intonetlon pettorne in a language la finite, juat like the 
elphobet. Once the ayatom hee been meetered it cen be epplled in one'a 
future performance, thua giving it fundementally an infinite acope. 

Thirdly, e devlent pronuncietion meene thet one ie immediately 'mark- 
ed' ebroed. A 'foreign accent' ie often ueed aa the meene to dlatingulah 
non-netivoe from netive epeekere. There cen be eome dlecuaalon ebout the 
propoeel that foreignere ehould cherieh their role ee non-notlvea. How- 
ever, one cen think of leernere who would prefer e netive-like pronuncie- 
tion. for example thoee with e pertlcularly atrong integrative motivetlon. 
end one cen imegine aituationa in which one wlehoa to evoid immodlete 
identification aa a foreigner. 

If pronunoielion ie worth leeching, it ie worth evalueting. For this 
evaluation one needa n apeech aample of tho teetee. Thie cen be obtained 
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toy involving him in a more or last epontaneoua convocation or by taking 
him queetiona about a text ha haa road btforo» about hia hobble a, hia 
piano for the future, etc* Howaver, it ia difficult to ecparate pronunciation 
from other tap act a of the teatce's orel production* auch at fluency , vocab- 
ulary, and grammatical accuracy f and Judge it independently (Clark and 
Bmnton 1810) • 

If th* tMting fooui ia on pronunciation, it ia a common practice to 
have the teatea read aloud a taw tinea or paragraphs of a written text. 
The advantage of this procedure ia the'* ona can obtain comparable apeech 
simples from different taeteoa, It cat m% be denied » howavar, that thta 
allcitatton procedure ovokaa an ur, Known variable: the ability to read 
aloud* The impact of thia variable ia unclear, Aa far aa we knowt no 
relevant research haa been carried out in order to determine it a effect. It 
could ceuae aevere probleme with taeteee who are to eome extent dyalectte. 

Nevertheless, reading aloud la very common in language teaching and 
uaually teateea are familiar with it. Moreover t we cannot neglect the ad- 
vantage of obtaining comparable apeech samples. Having thare collected ie 
neceeeary in order to determine the quality of taating procedures, 

Normally, rating ia done by giving the teetee a mark for the overall 
impreaaion of hie pronunciation. Rating can ba baaed on a four-, Ave- or 
aix~polnt-acsla. Every point on the ecele ie defined by a description of one 
or two prominent charactoriatlca of the taatee'a performance, like 'occa- 
alonal phonemic errora, but generally comprehenaible 1 or 'many phonemic 
errore: very difficult to perceive meaning' (Clark 1071:93). The eeleotion 
of a certain mark or acale point can be both aupported and motivated by a 
written record made during tha teetee'e performance in which pronunciation 
errore ore aucceaalvely noted down. The accountability and tranaparency 
of auch a rating proceduro can be improved by working out auch records 
In a ayatematioal way and by letting the final mark depend on It ontlrely. 
Furthermore, o systematic and complete record can be uaed for diagnostic 
purposes. Weak points in a taateo'a pronunciation can bo subjected to 
aubaequent training, Propoaala for aettlng up auch records are found In 
aeveral taating guidelines (eg. Ilarria 1069:86-87, Heaton 1979:84-66, and 
Medaen 1983:66-66), Theae procedurea have in common that in a running 
text or in a Hat of unconnected sentences possible pronunciation problems, 
Including stress snd intonstlon problems* ere marked: for examplo (Harris 
1069:86). 

While Mr. Brown read his newsp aper, prlmsry streaa 

his wife finished pecking flTolo thea voiced final consonant(s) 
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for tht trip. Tht suitcaae was al- 
-ready quite full, and aha waa having vowel quality 

• gwafdetf ol difficulty finding primary atreas 

An unmarked vereion of tht ttat la read aloud by tht toatet and tht teater 
ttoka off tht plact a where an trror ia made. 



*• Statement of tha problem 

We decided to examine the reliability of a rating procedure for the evalu- 
ation of pronunciation by which only marked elemonte in a text read aloud 
art Judged in a rather atraightforward way: the realisation of each element 
could be Judged either aa correct or aa incorrect* The reliability of thla 
procedure waa to be compared with the reliability of the common practice 
when an overall imprtaaion mark ia given for the quality of the teatee'a 
pronunciation after reading a text aloud. We will call the first, the expe- 
rimental method* 'atomletio' and tha scoond, the traditional method, •hol- 
istic'. 
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aa. The Pretest 



First of alii an inventory waa made of common pronunciation error* in 
Gsrmsn and French aa spoken by Dutch people. Such inventortea can be 
extracted from aeveral textbooks for the teaching of the pronunciation of 
German and French, Two texts were aelected and adapted slightly. On an 
average, every 5th or «th word (German) and every 8th or Oth word 
(French) contained s pronunciation problem. The German text waa rend 
oloud by 10 adults, the French text by 14, All testeoe had learned the 
respective foreign lsngusge in tho secondary school. The tosteea' tspe 
recordings were marked by s psnel of fully qusllfled toschers, 7 for both 
Germon and French, after s short instruction. The tepo recordings were 
presontod only once, without stops or replsys. The retort wcro to mark 
the separate items oh s rating aheet of the type that waa outlined above. 
Only Incorrect reall«atlona of elements in the texts were msrkod, so that 
for each toatee and rater the number of pronunclstion errors could be 
cslculsted 

However, unmsrked items are ambiguous with regard to thoir interpreta- 
tion • An unmarked item can mean two thlnga; 
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<i> no pronunciation error haa been made* or . 

it) a pronunciation error haa not booh rogtatered. 

Itama that ara hardly marked by the ratara at aU ara of no tntereat. The 

teeteoe' pronunciation la either correct or arrora ara not evtdant enough* 

Such itama ara superfluou* If tha teatece ere to be discriminated on tha 

basia of tha quality of their pronunciation. Itama that ara marked aa 

inoorract to a eubetentiai axtant ara more important in thli respect. 

For a good item, te, an Item with a certain discriminative power, 
registration of arrora la a necessary, but not a sufficient, condition, it ta 
alao nacaaaary that a vaat majority of ratara ahould raaot In tha aama tray 
on tha aama Item and tha aama tea tee. Ideally, every ratar ahould mark an 
item aa Inoorract aaph ttma that It la raaltsad inoorrectly and do nothing If 
It ia realised in a correct way, in that eaec the procedure ia absolutely 
reliable. However i aa a rule abaoluta reliability cannot be attained. Faotora 
auch aa lack of power of perception or flagging concentration among the 
ratara will prevent this, Nevertheleaa a certain minimal requirement of 
reliability muat be met If the rating procedure ia to be worthwhile. 

In order to select aucoesaful itama for the testing procedure under 
In vaat 1 gat Ion we aat the following conditions: 

(1) the majority of ratera (8 out of ?> ahould register an error made by a 

certain teatee on a aped Ac item; 
(t) the ratara ahould ahow a certain meaaure of agreement in their ratlnga 

of all the teateee on a particular item, A minimum value of ,70 for an 

interrater reliability coefficient waa aat, 
(Aa an index for agreement we uaad percentage of agreement uncorrected 
for chance,) About 301 of the German and French itama mat both criteria. 

There are reasons to believe that there ta aome systematic difference 
between successful and uneuooeaaful itama. In general, successful items do 
not allow gradea of correctneaa In their realisation. They are either rea- 
lised correctly of Incorrectly but not •more or leaa' oorreotly , For exetnpls, 
the realiaatton of la either correct or Inoorract , whereaa a vowel can 
be more or leaa diphthongised or neaaHsed, Hera toleranoe can vary among 
ratera. in the aama way the aaplration of a unvoiced plosive can ba more 
or leaa pronounced. There le no clear cut-off point. 

Both teata were revised; the main difference between the resulting 
German and the French teat waa that the first waa revieed in a atrlotly 
mechanical way: only pronunciation prcbloma represented by itama that 
paaaed tha preteat ware selected, whereaa the French teat waa sd Justed 
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% " For th* trtil of the definitive t**t* mm up* recording* were mad*. Th* .. ,^ 
"f !'' text wa* wad rioud by pupil* in ***ond*ry echoole who hod learned Carman 

for 1*4 yeare *nd French for i-S y*ar*. ? 

ThMt war* 16 t.atw* and M it*m* for German *nd IB teateea and 40 
Item* for French. Th* tap* recording. war* rated by 14 fully qualified 
teacher* of 0*rnan and it fully quettftod tuohara of Praneh. again aftar a 
. /• • .hort instruction. Th* rriing prooadur* waa carried out in th* aaoa way 

" th. data anriyri. w« to aauoat. th. reliability of thl. £ 

propoaad .toaUtic prooadur*. RaliablUty can b* axpraaaad in a eocfflciant 
batwaan 1 and 0i 1 maana that th. t**t **..ura. tha dlffarancaa b*tw**n 
th. taataaa in a vary ayatwatieal, p*rf*ct way. thara ia ccmpl*t* agree- 
ment ..ong rat.ra. By.ry it., put. th. t..t*a. In th. came rank order 
from high.r to lower ability. 0 m**n* th.t th. t..t I. compl.t.ly unralUbl.. 
that there I. no .yatamatical vart.no* at all. 

Tha following reliability oo.fflol.nt. were found, ualng generall.ablllty 
thaory (Bolu. at .1. Brann.n IMS). 

' French Carman 

1 ratar/ •«« * M 

a ratara .M 



3.9, Setting a orttarlon 

5o far w* h.v. only oomp.rad tha raaulta of th. French and Carman 
.xp.rim.ntri Lata. It would be helpful to know If the new procedure would 
entail an improvement of trodltional practice. In order to an.w.r thl. 
queetlon th. tape reeordlnge were played once more and tha rater, were 
.eked to giv tho toeteea a traditional lmpraarion mark on a twenty-point 
•a* (that la: tha tan-point ado with half polnta aa la commonly uaad In 
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the Netherlands), This time t the following gcneroUsabiiity coefficients wore 

founds . : 



A comparison of thtM figures with those of tht atomistic tests show* that 
tht French tttt U superior to holiotto rating and that tho Gerroen teat it 

not. 



1.4. Preliminary conolutlona and discussion 

There seems to bo a chinos that traditional praotlco in ths tasting of 
pronunciation can be improved if an atomistic ttst is used. Such a tsst csn 
further ths tranapsrtnoy and accountability of ths svaluational procedure* 
It should consist of sbout 40 items snd bt dsslgnsd on ths basts of rec- 
orded performances. With sn invsntory of pronuncistion problems ss s 
stsrting point ons should try to find regular pitfall a in a text oa it is 
spoksn , 

In this study ws focusssd on ths relisbility of s rsting procedure that 
tried to make the evaluation of pronunciation more explicit and trsnspsr- 
ont. 

It should bs stresssd, however, that there could be fsctors involvsd 
in s holistic rsting thst ere not covered by the atomistic testing 
procedure, such ss prosody, livsUness, snd ovsrall articulation. Therefore 
we decided to csrry out furthsr resssroh into the validity of ths 
procedure. And, lest but not leaat: ths vslidity of svslusting 
pronuncistion on the basis of s tsxt thst is reed cloud hss not been 
determined yet* 



3. Validity study 

In order to dot ermine the vslidity of ths stomistio procedure, s new exper- 
iment hss bsen set up for French* In this experiment wo tried to simulate 
the regular procedure of a teacher svslusting ths pronuncistion of his or 
her own pupils. One csn conceive of three different ways of Judging 
pronunciation in such a 'natural' setting: 



French 



German 
•IS 
•71 



i rate**/ 
I raters 



•II 
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reading 
•loud 



atomistic 



The rating of apontanaoua epecch is considered most valid for there ia 
no interference with a tcatat'a abtHty to read aloud/ If thara ta a fairly 
atrong positive oorralation batwttn tha outoomaa of an atomiatio eooring of 
raadinf aloud and a holiatio rating of apontanaoua speech, than tha axpari~ 
mental mat hod ahould ba oonaidarad valid aa wall* If tha oorralation be- 
twtan holiatio ratinr of apontanaoua apaaoh, on tha ona hand, and holiatic 
rating of raiding aloud, on tha other, ia sufficiently high, than tha aa* 
aumption that reading aloud and apontanaoua apaaoh will yield a different 
quality of pronunciation ia falatflod, at laaat for our targot group, puplla 
in secondary education ona year bofore their final examination. 



3.1. Daaign 

The experiment was set up in such a way that 5 teachera of French each 
had to evaluate the pronunciation of 10 of their own pupils , The pupils' 
voices were tape-recorded. First, the teschers listened to lu testees 
reading s text aloud. They had to note down s global impression mark for 
each testee'a quality of pronunciation. Then, they had to judge tho ssme 
testees reeding the ssme text according to the experimental atomiatio 
method. Next they heard aamples of French spoken spontaneously by the 
testees. Thoy wore, however, put in a different sequence in order to 
reduce a possible recall effect. 

Samples of spontsneous speech wsre elicited by v6 le~plsying. The 
testae wss involved in a situstional dialogue, with a reel native apeaker, 
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playing the part of • foreign ae^ueintanea who could piovtde e hoUday Job 
or • poHbt Officer who hid to eooe with the piobleme of e atolen bicycle. 
- ^Agtlo they were oikod to not* doih • floWl toproiiten mark. In 
erd 9 r to aaaeae the reliability of tho oencurrent procedure tho complete 
cydo had to bo ropottod with tho mm tuchere end the e»rae teat we but 
with Brother text ond another eaaple Of epohtaneoua apeech. Thle le ■ 
either etrong form of reliability aaeeeimint. It to eatimeted by comparing 
parallel teetforrae. Tho method wee not restricted to o eimple teet-reteet 
proeoduro with tho eeme toot. 

The following roUability oooffldtnto woro found: 



1 


.73 < 


.84 < 


.81 


2 


.68 < 


.78 < 


.88 


3 


.75 



(KB 80) 



From thcae reaulte it followe that tho three tut forraeta iro approxl- 
raetely equivalent if we texe tho uaual 85% confidence llmita into account. 

llolietic rating a« will u itomiotio ecoring iro proved to bt quite 
roltibli. Now, if there ia a flirty atrong poaitWe correlation betwwn both 
procedure!, then atomiatio ecoring and holiatio rating maaauri thi aame 
variable. The following correlation coefflclente were found (again with 851 
confidence limita): 





1 
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1 


,7i < .84 < .80 


.87 < 


.61 < 


.70 


2 




.86 < 


.58 < 


.74 



The correlation between holietie rating end etoenietic ecoring are only 
moderate, whereee the correction between holiitto reting of apontineoua 
apeech and reading eloud la quite high. 



3.2. Conclusion 

On the beeia of thie validity etudy the following conolueions can bo dre 
in the upper forma of eeeondery eehooJ one cen give value Judgemente 
pronunciation by hevtng the teeteee read aloud a text, ee well oa on 
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'ba'eie of • aempia of epdnteneoua apeooh, Furthermore, •tomiotlo acorlng It 
not a vary good alternative for heiletic rating* it cannot predict a general 
imprcealon of tha quality of a teetee'a pronunciation. Beetde* aegmentel 
featurea and atraaa or intonation pattorha thart «uet bt ono or two othor 
relevant fectora In it that determine tha quality Of pronunciation. 

Natural rhythm, Hvellneea, apeaking rata? We do not know yat axactly 
what thay arc. Further raaaaroh haa to ba carried out. 
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